Sketchy Inference: Towards Streaming LDA

Recent developments in inference algorithms based on stochastic Expectationmaximization or stochastic cellular automata (SCA) have made it possible to employ a variety of randomized data structures that are unavailable to the dominant inference methods in the Bayesian toolkit, including collapsed Gibbs sampling and stochastic variational inference (SVI). Equipped with this recent capability, we make progress towards a true streaming inference algorithm for LDA that makes novel use of these random data-structures. We mean “true” in the sense that the algorithm avoids entirely the need for pre-computation, which many current “streaming” variants of latent Dirichlet allocation (LDA) must perform. We find that despite using various randomized data-structures to represent the sufficient statistics, the inference algorithm converges to similar perplexities as more conventional LDA while producing equally interpretable topics.